Reducing write i/o latency using asynchronous fibre channel exchange

ABSTRACT

Methods and systems are provided for Reducing Write I/O Latency Using Asynchronous Fibre Channel Exchange. A FCP target device may send one or more FC write control information units (IUs) to a FCP initiator device within a first FC exchange to request a transfer of data associated with a FCP write command IU previously sent to the FCP target device by the FCP initiator device within a second FC exchange. The FC write control IUs may be sent without the FCP target device first receiving from the FCP initiator device a sequence initiative of the second FC exchange, and/or may be sent within the first FC exchange concurrently with the FCP initiator device sending one or more FCP data IU sequences within the second FC exchange to the FCP target device. Thus, a full-duplex communication environment may be setup between the FCP initiator device and FCP target device.

BACKGROUND

The following is an excerpt from the Introduction to the AmericanNational Standard for Information Technology-Fibre Channel Protocol forSCSI (FCP), Fourth Version (FCP-4), approved Oct. 12, 2011, INCITS481-2011, which is incorporated by reference herein in its entirety forall purposes.

-   -   The Small Computer System Interface (SCSI) command set is widely        used and applicable to a wide variety of device types. The        transmission of SCSI command set information across Fibre        Channel links allows the large body of SCSI application and        driver software to be successfully used in the high performance        Fibre Channel environment.    -   This standard describes the protocol for transmitting SCSI        commands, data, and status using Fibre Channel FC-FS-3 Exchanges        and Information Units. Fibre Channel is a high-speed serial        architecture that allows either optical or electrical        connections. The topologies supported by Fibre Channel include        point-to-point, fabric switched, and arbitrated loop. All Fibre        Channel connections use the same standard frame format and        standard hierarchy of transmission units to transmit the        Information Units that carry SCSI information.

The American National Standard for Information Technology-Fibre ChannelFraming and Signaling (FC-FS-3), approved Dec. 28, 2011, INCITS470-2011, is also incorporated by reference herein in its entirety forall purposes.

A conventional SCSI write command according to the FCP standard involvesthe following steps, which are described in more detail below. First, aFCP initiator sends a SCSI write command to a FCP target encapsulated ina FCP command frame. The target responds with a transfer ready message,which is a data delivery request, in which the target indicates to theinitiator the amount of buffer space available to receive the writedata. The initiator responds by sending the amount of data the targetindicated it could receive. The target then sends another transfer readymessage again indicating the amount of buffer space available and theinitiator sends more data. This back and forth process repeats until allthe data has been transferred and the target returns status to theinitiator to complete the command. Due to current limitations of the FCPstandard that will now be described, the initiator must effectively waitto send the data until it has received the relevant transfer readymessage, and the target must wait to send (all but the first of) thetransfer ready messages until it has received the data associated withthe previous transfer ready message. This increases the latency of theFCP write I/O operation.

The FCP defines an Exchange as the basic mechanism that transfersinformation consisting of one or more related non-concurrent Sequencesthat may flow in the same or opposite directions. The Exchange isidentified by an Originator Exchange_ID (OX_ID) and a ResponderExchange_ID (RX_ID). The FCP defines a Sequence as set of one or moredata frames with a common Sequence_ID (SEQ_ID), transmittedunidirectionally from one N_Port to another N_Port with a correspondingresponse, if applicable, transmitted in response to each data frame.Thus, an Exchange between a FCP initiator port and a FCP target port maybe viewed as a half duplex operation. That is, the Sequences within anExchange can be transferred in only one direction at a time between. Theport that is authorized to send a Sequence in a given point in time isreferred to as holding the Sequence Initiative. In a typical FCP writeI/O operation, the initiator transfers the Sequence Initiative to thetarget in the last data frame of the Sequence. The target responds withthe transfer ready message, which transfers the Sequence Initiative backto the initiator so it can send more data frames (or the status frame).Thus, a half duplex handshaking occurs via the passing back and forth ofthe Sequence Initiative.

The FCP defines an Information Unit (IU) as an organized collection ofdata specified by the Fibre Channel Protocol to be transferred as asingle Sequence by the Fibre Channel service interface. The FCP standardmaps a SCSI I/O operation, such as a SCSI I/O write operation, into asingle Fibre Channel Exchange, which means the IUs are transferredbetween the initiator and target in non-concurrent Sequences, that is,in a half duplex manner, as described above. Thus in the above example,while the data IU is being transferred from the initiator to the target,the target is not permitted to send another transfer ready IU to theinitiator to notify the initiator of additional buffer space until theinitiator transfers the Sequence Initiative to the target, even thoughthe buffer space may have become available in the target well before theinitiator transfers the Sequence Initiative to the target. Consequently,there is latency introduced into the FCP write I/O operation because ofthe half-duplex nature of the single Exchange in which the target andinitiator perform the write I/O operation.

More specifically, there are multiple components of the latencyintroduced by the half-duplex nature of a single Exchange. First, thereis the transmission time to transmit from the initiator to the targetover the FC fabric the data frame that transfers the Sequence Initiativefrom the initiator to the target. The transmission time includestransmission medium delay (e.g., copper wire or fiber optic cablepropagation delay) as well as any delay introduced by switches in the FCfabric along the path between the target and initiator. Second, there isthe target processing time from when the target receives the SequenceInitiative-transferring data frame until it transmits the transfer readyIU, which transfers Sequence Initiative back to the initiator. Theprocessing time is taken by the target hardware and/or firmware toprocess the Sequence Initiative-transferring data frame to determinethat the target now has the Sequence Initiative to transmit the transferready IU. Third, there is the transmission time to transmit the transferready IU from the target to the initiator over the FC fabric. Fourth,there is the initiator processing time from when the initiator receivesthe Sequence Initiative-transferring transfer ready IU until ittransmits the first data frame of the next Sequence. This is time takenby the initiator hardware and/or firmware to process the SequenceInitiative-transferring transfer ready IU.

The sum of these latencies introduced by the half-duplex nature of thesingle Exchange in which the target and initiator perform the write I/Ooperation has been observed to be on the order of tens of microsecondsin some cases, which may significantly reduce performance of the system.Where the target is a mechanical hard disk drive, tape drive or otherperipheral device having a relatively large access time (e.g., rotationlatency and/or seek times on the order or milliseconds), the half-duplexExchange-induced latency may have been small relative to the peripheralaccess time when viewing the entire write I/O operation time. However,with the advent of new low access time peripherals, such as solid-statedisks (SSDs), and considering cases of high peripheral cache hit rates,the half-duplex Exchange-induced latency has become even moresignificant. Furthermore, longer cable distances through the FC fabricbetween the target and initiator may exacerbate the latency,particularly the transmission times, as may longer paths through the FCfabric, e.g., due to larger number of switch hops. Finally, large I/Owrite sizes may further exacerbate the latency, particularly in cases inwhich a relatively large number of transfer ready IUs must be sent in agiven Exchange.

BRIEF SUMMARY

Embodiments are described that provide an additional FC Exchange,referred to as an asynchronous write control Exchange (AWCE), thatenables some of the latencies associated with the conventional singlehalf-duplex FCP write I/O Exchange to be reduced or eliminated. Thetarget sends write control IUs (for example, enhanced transfer ready FCPIUs or a new FC IU previously undefined by the FCP-4) to the initiatoron the AWCE in a pipelined fashion to notify the initiator that thetarget is ready to receive data (e.g., buffer space is available)without having to wait for the initiator to transfer the SequenceInitiative of the Exchange originated by the initiator sending the FCPwrite command IU. In this way, the FCP write I/O operation has afull-duplex nature due to the AWCE in combination with the Exchangeoriginated by the initiator, which may significantly reduce the latencyfor a single write I/O operation as well as overall performance of asystem performing many such write I/O operations.

In one aspect embodiments provide a Fibre Channel (FC) Protocol for SCSI(FCP) target. The FCP target includes a FC port and a processor adaptedto communicate with a FCP initiator via the FC port. The FCP target isconfigured to send one or more FC write control information units (IUs)to the FCP initiator within a first FC exchange to request a transfer ofdata associated with a FCP write command IU previously sent to the FCPtarget by the FCP initiator within a second FC exchange. The first FCexchange is distinct from the second FC exchange. A payload of each ofthe one or more write control IUs includes an originator exchangeidentifier (OX_ID) value with which the FCP initiator originated thesecond exchange and a responder exchange identifier (RX_ID) valueassigned by the FCP target for the second exchange.

In another aspect embodiments provide a Fibre Channel (FC) Protocol forSCSI (FCP) initiator. The FCP initiator includes a FC port and aprocessor adapted to communicate with a FCP target via the FC port. TheFCP initiator is configured to receive one or more FC write controlinformation units (IUs) from the FCP target within a first FC exchangerequesting a transfer of data associated with a FCP write command IUpreviously sent to the FCP target by the FCP initiator within a secondFC exchange. The first FC exchange is distinct from the second FCexchange. A payload of each of the one or more write control IUsincludes an originator exchange identifier (OX_ID) value with which theFCP initiator originated the second exchange and a responder exchangeidentifier (RX_ID) value assigned by the FCP target for the secondexchange.

In yet another aspect embodiments provide a method that includes a FibreChannel (FC) Protocol for SCSI (FCP) target sending one or more FC writecontrol information units (IUs) to an FCP initiator within a first FCexchange to request a transfer of data associated with a FCP writecommand IU previously sent to the FCP target by the FCP initiator withina second FC exchange. The first FC exchange is distinct from the secondFC exchange. A payload of each of the one or more write control IUsincludes an originator exchange identifier (OX_ID) value with which theFCP initiator originated the second exchange and a responder exchangeidentifier (RX_ID) value assigned by the FCP target for the secondexchange.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system comprising a FibreChannel (FC) network according to an embodiment.

FIG. 2 is a flow diagram illustrating the flow of FC frames between aFCP initiator and a FCP target according to a conventional FCP write I/Ooperation.

FIG. 3 is a block diagram illustrating the payload of a write_control IUaccording to an embodiment.

FIG. 4 is a flow chart illustrating operation of a FCP initiator to senda FCP write command IU to a FCP target 104 according to an embodiment.

FIG. 5 is a flow chart illustrating operation of the initiator totransfer data to the target associated with a FCP write command IU assent according to FIG. 4 according to an embodiment.

FIG. 6 is a flowchart illustrating the origination of an AWCE.

FIG. 7 is a flowchart illustrating operation of the target to sendwrite_control IUs.

FIG. 8 is a flow diagram illustrating the flow of FC IUs between a FCPinitiator and a FCP target of the system of FIG. 1 to perform a FCPwrite I/O operation according to the operation of the initiator andtarget described with respect to the flowcharts of FIGS. 4 through 7.

FIG. 9 is a flow diagram illustrating one embodiment of the manner inwhich an initiator and a target perform multiple write I/O operations.

FIG. 10 is a flow diagram illustrating an alternate embodiment of themanner in which an initiator and a target perform multiple write I/Ooperations.

FIG. 11 is a block diagram illustrating the payload of a write_controlIU according to an alternate embodiment.

FIG. 12 is a block diagram illustrating the payload of a write_controlIU 1200, which is an enhanced FCP_XFER_RDY IU, according to an alternateembodiment.

FIG. 13 is a timing diagram illustrating latencies associated with aconventional FCP write I/O operation such as described in FIG. 2.

FIG. 14 is a timing diagram illustrating latencies associated with a FCPwrite I/O operation such as described in FIG. 8.

DETAILED DESCRIPTION OF THE EMBODIMENTS Glossary

A FC port is a FC link control facility (LCF) that includes atransmitter and a receiver for transmitting and receiving Fibre Channelframes on a FC link. A FC port has a distinct address identifier.

A FC write control information unit (IU) is one or more FC frames sentby a FCP target to a FCP initiator to request a transfer of dataassociated with a FCP write command received by the FCP target.

A FC Exchange is the basic mechanism that transfers informationconsisting of one or more related non-concurrent FC Sequences that mayflow in the same or opposite directions. An Originator Exchange_ID(OX_ID) and a Responder Exchange_ID (RX_ID) identify the Exchange. A FCSequence is a set of one or more FC data frames with a commonSequence_ID (SEQ_ID), transmitted unidirectionally from one FC N_Port toanother N_Port.

A FCP initiator is a FCP node that sends FCP command IUs to a FCPtarget. An example of FCP an initiator is a FCP host bus adapter.

A FCP target is a FCP node that receives FCP command IUs from a FCPinitiator. Examples of FCP targets include storage devices, such as diskdrives, solid state disks (SSDs), tape drives, CDROM drives, and thelike.

A FCP write command IU is a FCP command (FCP_CMND) IU that has theWRDATA bit of the payload set to a value of one to indicate theinitiator expects to transmit FCP data (FCP_DATA) IUs to the target. Itshould be understood that a FCP write command IU is not limited to aFCP_CMND IU that encapsulates a SCSI WRITE command, but instead includesother SCSI commands that implicate writing data from the initiator tothe target.

Referring now to FIG. 1, a block diagram illustrating a systemcomprising a Fibre Channel (FC) network 100 according to an embodimentis shown. The network 100 includes a FC fabric 106 connecting aplurality of FCP initiators 102 and FCP targets 104, referred togenerically as FC nodes 102/104. Each FC node 102/104 includes a FCN_Port 112, a memory 114 and a CPU 116. The N_Ports 112 are coupled toFC ports (not shown) of the FC fabric 106 via FC links. The FC links arepairs of unidirectional transmission mediums for transmitting inopposite directions, such as copper wire or optical fiber cables asdescribed in the FC specifications or other transmission media that maybe subsequently developed and employed in a FC network. The FC fabricmay include FC switches or other entities that interconnect the N_Portsand are capable of routing FC frames using the information in a FC frameheader.

The N_Port 112 of each node 102/104 includes a Link Control Facility(LCF) for transmitting and receiving FC frames. The N_Port 112 mayinclude intelligence, in the form of hardware and/or software, toprocess incoming and outgoing FC frames. The memory 114 includes bufferspace for buffering FC frames. The N_Port 112 receives FC frames fromthe memory 114 for transmission on the FC link and writes to the memory114 FC frames received from the FC link. The CPU 116 may process FCframes for sending and receiving by the N_Port 112. Additionally, theCPU 116 may manage the buffer space in the memory 114. The variousactions described herein performed by the FC nodes 102/104 includingconventional FC node 102/104 operation as well as the enhancedasynchronous write control exchange (AWCE) capability may be performedby the N_Port 112, the CPU 116 or a combination thereof.

Referring now to FIG. 2, a flow diagram illustrating the flow of FCframes between a FCP initiator and a FCP target according to aconventional FCP write I/O operation is shown. The initiator sends tothe target a FCP_CMND IU 202 that specifies a command for which theinitiator writes data to the target. The FCP_CMND IU 202 originates anFC Exchange within which the write I/O operation is performed by theinitiator and target in a half-duplex fashion as described above anddescribed in more detail here. The frame header of the FCP_CMND IUspecifies an originator exchange identifier (OX_ID), which in theexample of FIG. 2 has a value of 0x1111; a responder exchange identifier(RX_ID), which in the example of FIG. 2 has a value of 0xFFFF; and aSequence Initiative (SI) bit, which in the example of FIG. 2 has a valueof one to cause the Sequence Initiative to be transferred to the target.It should be noted that if the FCP_CMND IU is to utilize the FCP FIRSTBURST feature, then the initiator retains the Sequence Initiative aftersending the FCP_CMND IU.

In response to the FCP_CMND IU 202, the target sends the initiator aFCP_XFER_RDY IU 204. The frame header of the FCP_XFER_RDY IU 204specifies the OX_ID value received in the FCP_CMND IU, which in theexample of FIG. 2 has a value of 0x1111; a RX_ID assigned by the targetwhich the initiator learns, which in the example of FIG. 2 has a valueof 0x2222; and a SI bit, which in the example of FIG. 2 has a value ofone to cause the Sequence Initiative to be transferred back to theinitiator.

In response to the FCP_XFER_RDY IU 204, the initiator sends the target aFCP_DATA IU 206. The frame header of the FCP_DATA IU 206, as do theremainder of the FC frame headers sent within the FC Exchange describedin FIG. 2, specifies the OX_ID value of 0x1111 and the RX_ID value of0x2222. The SI bit has a value of zero to cause the Sequence Initiativeto be held by the initiator for all but the last FC frame of theSequence, which has a SI bit value of one to cause the SequenceInitiative to be transferred back to the target.

In response to the last frame of the FCP_DATA IU 206, the target sendsthe initiator another FCP_XFER_RDY IU 208 similar to IU 204. However,the second FCP_XFER_RDY IU 208 has different FCP_DATA_RO andFCP_BURST_LEN values than the first FCP_XFER_RDY IU 204. Importantly,according to conventional operation, the target does not send theFCP_XFER_RDY IU 208 until the FCP_DATA IU transfers the SequenceInitiative to the target due to the half-duplex nature of the single FCExchange in which the conventional write I/O operation is performed.

In response to the second FCP_XFER_RDY IU 208, the initiator sends thetarget a second FCP_DATA IU 212, similar to the first FCP_DATA IU 206but having different data in its payload.

In response to the last frame of the FCP_DATA IU 212, the target sendsthe initiator a FCP_RSP IU 214 that includes, among other things, theSCSI status associated with the write I/O operation. Typically via theFCP_RSP IU 214, the target terminates the FC Exchange that wasoriginated by the initiator via the FCP_CMND IU 202.

As may be observed from FIG. 2, the various latencies described aboveand below with respect to FIG. 13 may be incurred during a write I/Ooperation performed according to the conventional half-duplex single FCExchange paradigm.

Referring now to FIG. 3, a block diagram illustrating the payload of awrite_control IU 300 according to an embodiment is shown. Thewrite_control IU 300 may have in its R_CTL field of its FC frame headera value that is currently unused by the FS-FS standard (for example,0x04) to distinguish it from other IUs, such as the FCP_XFER_RDY(however, see FIG. 12 for an alternate embodiment that employs anenhanced FCP_XFER_RDY IU as a write_control IU 1200). The write_controlIU 300 payload includes an OX_ID field 302, an RX_ID field 304, anoperation _type field 306, a sequence _initiative field 308, arelative_offset field 312 and a transfer_length field 314.

The target 104 populates the OX_ID field 302 with the value of the OX_IDfield of the header of the write command IU (FCP_CMND) received from theinitiator 102 that originated the write I/O Exchange (WIOE). The WIOE isa different FC Exchange than the FC Exchange within which the target 104sends the write_control IU 300 to the initiator 102, which is referredto as an asynchronous write control Exchange (AWCE). The WIOE and AWCEin combination form a full-duplex mode of communication between theinitiator 102 and target 104. This enables the target 104, even thoughit does not have Sequence Initiative on the WIOE, to notify theinitiator 102 within the AWCE that the initiator 102 can send dataassociated with the write I/O, as discussed in more detail below.

The target 104 populates the RX_ID field 304 with a value chosen by thetarget 104 that enables the initiator 102 to learn the RX_ID value ofthe WIOE. In conventional operation, the initiator 102 would learn theRX_ID value of the WIOE within the WIOE, that is, from a FC frame sentby the target 104 to the initiator 102 within the WIOE. However,according to the embodiments described herein, the initiator 102 learnsthe RX_ID value of the WIOE within the AWCE rather than within the WIOE,as described in more detail below. In this way, the OX_ID field 302 andRX_ID field 304 enable the initiator 102 to identify the write commandIU for which the payload of the write_control IU 300, namely therelative_offset 312 and transfer_length 314 fields, is requesting a datatransfer. The initiator 102 populates the OX_ID and RX_ID fields of theFC header of each FC frame of the FCP_DATA IU sent within the WIOE withthe values provided by the target 104 in the OX_ID field 302 and RX_IDfield 304, respectively, of the write_control IU 300.

The operation_type field 306 indicates the type of operation beingrequested by the write_control IU 300. In one embodiment, a value ofzero indicates a request_dataout_transfer, that is, a request for theinitiator 102 to transfer data of the associated write command IU to thetarget 104, similar to a FCP_XFER_RDY IU. Other values of theoperation_type field 306 are reserved or may be used as described inalternate embodiments below.

The sequence_initiative field 308 instructs the initiator 102 whether totransfer the Sequence Initiative of the WIOE to the target 104 (value=1)or to not transfer the Sequence Initiative of the WIOE to the target 104(value=0) via the last frame of the FCP data IU sent within the WIOE bythe initiator 102 to the target 104 in response to the write_control IU300. In one embodiment, the target 104 may set the transfer_length field314 to zero to allow the initiator 102 to send a data IU with nopayload, but which transfers the Sequence Initiative to the target 104,which may in turn allow the target 104 to send a FCP_RSP IU within theWIOE to the initiator 102 in cases of an error.

When the operation_type field 306 specifies a request_dataout_transfer,the relative_offset 312 and transfer_length 314 fields have similarsemantics to the FCP_DATA_RO and FCP_BURST_LEN fields, respectively, ofthe FCP_XFER_RDY IU. That is, the relative_offset field 312 specifiesthe position of the first byte of data requested by the write_control IU300 relative to the first byte of all the data transferred by the writecommand IU, and the transfer_length field 314 specifies the number ofbytes of data the target 104 is requesting, e.g., the amount of bufferspace the target 104 is prepared to receive in the next FCP_DATA IU.

FIGS. 4 and 5 describe operation of the initiator 102, FIGS. 6 and 7describe operation of the target 104, and FIG. 8 describes combinedoperation of the initiator 102 and target 104, according to embodiments.

Referring now to FIG. 4, a flow chart illustrating operation of a FCPinitiator 102 to send a FCP write command IU to a FCP target 104according to an embodiment is shown. Flow begins at block 402.

At block 402, the initiator 102 sends a FCP write command IU to thetarget 104. The write command IU originates a WIOE with the target 104.The write command IU specifies a SCSI command that involves theinitiator 102 writing data to the target 104. Specifically, the WRDATAbit of the FCP_CMND IU payload is set to a value of one to indicate theinitiator 102 expects to transmit FCP_DATA IUs to the target 104. Thewrite command IU is similar to a conventional FCP write command IU;however, unconventionally, preferably the write command IU retainsSequence Initiative of the WIOE. Advantageously, this enables theinitiator 102 to send data IUs to the target 104 within the WIOEconcurrently with the target 104 sending write_control IUs 300 to theinitiator 102 within the AWCE to reduce the latency associated with aFCP write IO operation, as described herein. The OX_ID field of thewrite command IU frame header will contain a value that the initiator102 guarantees to be unique with respect to the OX_ID of any other openFC Exchange with the target 104. The target 104 will remember the writecommand IU frame header OX_ID field and use the value to populate theOX_ID field 302 of the write_control IU 300 it sends at block 704 ofFIG. 7. Flow proceeds to block 404.

At block 404, the initiator 102 waits to receive from the target 104 awrite_control IU 300 that identifies the WIOE and requests a datatransfer from the initiator 102. In this way, the initiator 102 learnsthe RX_ID value of the WIOE. The initiator 102 knows that it willreceive the write_control IU 300 from the target 104 within a differentExchange than the WIOE, namely the AWCE. The origination of the AWCE isdescribed in more detail below with respect to FIG. 6. Variousembodiments are contemplated for the initiator 102 to know that it willreceive the write_control IU 300 from the target 104 within an AWCErather than within the WIOE at block 404 and for the initiator 102 toknow that it should not transfer the Sequence Initiative to the target104 when sending the write command IU at block 402. In a preferredembodiment, the initiator 102 and target 104 communicate with oneanother during an initialization time to determine whether they supportthe AWCE capability and only employ the capability if both support it.For example, the target 104 and initiator 102 may communicate using theprocess login (PRLI) or port login (PLOGI), which are described in theFC standards. Flow ends at block 404.

It should be understood that during operation of the system 100, theinitiator 102 may send multiple write command IUs to the target 104according to block 402, and multiple of the write command IUs may beoutstanding at a given point in time. Thus, multiple WIOEs between theinitiator 102 and the target 104 may be open at a given point in time.Further, it should be understood that the initiator 102 may send writecommand IUs to multiple targets 104 of the system 100, and the WIOEsbetween the initiator 102 and the multiple targets 104 may be open at agiven point in time. Finally, it should be understood that multipleinitiators 102 of the system 100 may send write command IUs to thetargets 104, and the WIOEs between the initiators 102 and the targets104 may be open at a given point in time.

Referring now to FIG. 5, a flow chart illustrating operation of theinitiator 102 to transfer data to the target 104 associated with a FCPwrite command IU as sent according to FIG. 4 according to an embodimentis shown. Flow begins at block 502.

At block 502, the initiator 102 receives a write_control IU 300 from thetarget 104 within an AWCE. The sending of the write_control IU 300 bythe target 104 within the AWCE is described with respect to FIG. 7below. The AWCE is a different FC Exchange than the FC Exchange withinwhich the initiator 102 sent the write command IU, which is referred toas the WIOE as described above with respect to FIG. 4. The originationof the AWCE is described below with respect to FIG. 6. The write_controlIU 300 identifies the WIOE within which the write command IU was sentand therefore identifies the write command IU for which thewrite_control IU 300 is requesting data. That is, the write_control IU300 identifies one of the write command IUs sent at block 402.Specifically, the write_control IU 300 OX_ID field 302 identifies theOX_ID of the WIOE, and the write_control IU 300 RX_ID field 304 valueenables the initiator 102 to learn the RX_ID value of the WIOE and touse the learned RX_ID value in the RX_ID field of the FC headers of thedata IU frames sent at block 504. The target 104 chooses the RX_ID valueof the WIOE at block 704 of FIG. 7. Flow proceeds to block 504.

At block 504, the initiator 102 sends a data IU (FCP_DATA IU) to thetarget 104 within the WIOE identified in the write_control IU 300received at block 502. That is, the initiator 102 populates the OX_IDand RX_ID fields of the header of each FC frame of the data IU with thevalues of the OX_ID 302 and RX_ID 304 fields, respectively, that werereceived in the write_control IU 300 at block 502. The data IU includesin its payload the data requested by the write_control IU 300 receivedat block 502 in its relative_offset 312 and transfer_length 314 fields.Preferably, the initiator 102 retains the Sequence Initiative of theWIOE. That is, the initiator 102 clears the Sequence Initiative bit tozero in each of the FC frames of the data IU sent to the target 104 inorder to hold the Sequence Initiative. This is a change from theconventional FCP operation in which the initiator 102 would transfer theSequence Initiative to the target 104 so that the target 104 could senda FCP_XFER_FDY IU to request more data, as described above with respectto FIG. 2. Advantageously, the initiator 102 need not transfer the WIOESequence Initiative to the target 104 because the target 104 is able toasynchronously request the next data via the AWCE. That is, the target104 is able to send the write_control IUs 300 to the initiator 102within the AWCE (concurrently with the initiator 102 sending the dataIUs within the WIOE) without having to wait for the initiator 102 totransfer Sequence Initiative of the WIOE to the target 104. Theinitiator 102 transfers Sequence Initiative of the WIOE to the target104 via the data IU only if: (1) all of the data for the write commandIU has been sent, that is, with this data IU the entire amount of thedata associated with the write command IU has been transferred; or (2)the target 104 notified the initiator 102 to transfer the WIOE SequenceInitiative to the target 104, for example via the sequence_initiativefield 308 of the write_control IU 300 received at block 502. Flow endsto block 504.

It should be understood that the initiator 102 performs the operationsof FIG. 5 as many times as necessary to transfer all the data specifiedby the write command IU sent at block 402. Similarly, the target 104performs the operation at block 704 of FIG. 7 described below as manytimes as necessary to request transfer of all the data specified by thewrite command IU received at block 702. The operations of FIG. 5 and theoperation of block 702 are purposely not shown in a loop and not shownin sequence with one another because the sending of the write_controlIUs 300 at block 702 and the sending of data IUs at block 504 mayadvantageously be performed concurrently with one another within therespective AWCE and WIOE in a full duplex fashion to reduce the latencyassociated with a FCP write command IU. That is, the target 104 may sendthe write_control IUs 300 within the AWCE asynchronously with respect tothe sending of the write command IUs within the WIOE by the initiator102. Stated alternatively, the target 104 is enabled to send thewrite_control IUs 300 within the AWCE without holding SequenceInitiative of the WIOE.

Referring now to FIG. 6, a flowchart illustrating the origination of anAWCE is shown. Flow begins at block 602.

At block 602, a target 104 originates a FC Exchange, an AWCE, with aninitiator 102 that supports the AWCE capability. Specifically, thetarget 104 clears to zero the Exchange Context bit of the F_CTL field ofthe write command IU frame header to indicate the target 104 is theOriginator of the Exchange, that is, of the AWCE. Preferably, the target104 originates the AWCE in response to a write command IU received fromthe initiator 102. As discussed in more detail below, the target 104 mayoriginate the AWCE in response to each write command IU received, suchas described below with respect to FIG. 10. However, in otherembodiments the target 104 may service multiple write command IUs usinga single AWCE, such as described below with respect to FIG. 9. In oneembodiment, the target 104 may employ the same AWCE for the entireprocess login session with the initiator 102. Furthermore, otherembodiments are contemplated in which the target 104 originates the AWCEat a time other than in response to a write command IU received from theinitiator 102, for example at an initialization time, such as a portlogin or process login. In such embodiments, the target 104 may populatethe operation_type field 306 with a value (other thanrequest_dataout_transfer) to indicate that the target 104 is originatingthe AWCE without respect to a particular WIOE, in which case the valueof the OX_ID field 302 will be ignored by the initiator 102. Suchembodiments may be particularly compatible with embodiments in which thetarget 104 services multiple write command IUs using a single AWCE.Still further, embodiments are contemplated in which the target 104originates and terminates multiple AWCEs within a given write I/Ooperation, such as each time it wants to send a write_control IU 300 ofmultiple write_control IUs 300 for a write I/O operation. Finally, otherembodiments are contemplated in which the initiator 102 originates theAWCE and subsequently transfers Sequence Initiative of the AWCE to thetarget 104. For example, the initiator 102 may send the target 104 awrite_control IU 300 with a value (other than request_dataout_transfer)of the operation_type field 306 to indicate that the initiator 102 isoriginating the AWCE. In this case, the initiator 102 will transferSequence Initiative to the target 104 after originating the AWCE.

It should be understood that during operation of the system 100,multiple AWCEs between the initiator 102 and the target 104 might beopen at a given point in time. Further, it should be understood thatAWCEs between the initiator 102 and multiple targets 104 might be openat a given point in time. Finally, it should be understood that AWCEsbetween multiple initiators 102 and multiple targets 104 may be open ata given point in time.

Referring now to FIG. 7, a flowchart illustrating operation of thetarget 104 to send write_control IUs 300 is shown. Flow begins at block702.

At block 702, the target 104 receives from the initiator 102 a FCP writecommand IU. The write command IU originates a WIOE, as described abovewith respect to block 402 of FIG. 4. Flow proceeds to block 704.

At block 704, the target 104 sends one or more write_control IUs 300 tothe initiator 102 within an AWCE to notify the initiator 102 that thetarget 104 is ready to receive data associated with the write command IUreceived at block 702. As described above with respect to block 502 ofFIG. 5, each write_control IU 300 identifies the WIOE within which thewrite command IU was received at block 702 and therefore identifies thewrite command IU for which the write_control IU 300 is requesting data.Specifically, the write_control IU 300 OX_ID field 302 identifies theOX_ID of the WIOE, and the write_control IU 300 RX_ID field 304 valueenables the initiator 102 to learn the RX_ID value of the WIOE and touse the learned RX_ID value in the RX_ID field of the FC headers of thedata IU frames sent at block 504. The target 104 chooses the RX_ID valueof the WIOE and guarantees it to be unique with respect to the RX_ID ofany other open FC Exchange with the initiator 102. The target 104 sendsthe one or more write_control IUs 300 to the initiator 102 one at atime. However, advantageously, embodiments described herein enable thetarget 104 to send the one or more write_control IUs 300 to theinitiator 102 within the AWCE concurrently with the initiator 102sending data IUs to the target 104 within the WIOE originated by thewrite command IU and asynchronously to the initiator 102 transferringSequence Initiative of the WIOE to the target 104, thereby reducing oreven eliminating many of the latencies associated with the conventionalFCP write I/O operation described above with respect to FIG. 2, forexample. An embodiment is also described with respect to FIG. 11 inwhich a single write_control IU 300 may request data for multiple writecommand IUs associated with multiple corresponding WIOEs. The target 104sends a write_control IU 300 each time it determines that it has not yetrequested all of the data associated with the write command IU and thatit is ready (e.g., has buffer space) to receive some of the unrequesteddata. Various algorithms are contemplated for determining the minimumamount of buffer space available before sending the write_control IU300. Flow ends at block 704.

Referring now to FIG. 8, a flow diagram illustrating the flow of FC IUsbetween a FCP initiator 102 and a FCP target 104 of the system 100 ofFIG. 1 to perform a FCP write I/O operation according to the operationof the initiator 102 and target 104 described with respect to theflowcharts of FIGS. 4 through 7 is shown. Two different FC Exchanges areshown in FIG. 8: a WIOE and an AWCE. FC IUs sent within the WIOE areshown within rectangular boxes having square corners, whereas FC IUssent within the AWCE are shown within rectangular boxes having roundedcorners. As may observed from FIG. 8 and as discussed more below, thesending of at least one write_control IU 300 by the target 104 to theinitiator 102 within the AWCE is asynchronous to and concurrent with thesending of FCP data IU frames by the initiator 102 to the target 104within the WIOE.

As described at block 402 of FIG. 4, the initiator 102 sends a writecommand IU (FCP_CMND) IU to the target 104, which originates the WIOE.The frame header of the write command IU specifies an OX_ID that in theexample of FIG. 8 has a value of 0x1111, and a RX_ID that in the exampleof FIG. 8 has a value of 0xFFFF. Preferably, the write command IU doesnot cause the Sequence Initiative of the WIOE to be transferred to thetarget 104.

As described at block 704 of FIG. 7, in response to receiving the writecommand IU, when the target 104 is ready (e.g., buffer space isavailable) to receive data specified by the write command IU, the target104 sends the initiator 102 a write_control IU 300 within an AWCE.According to one embodiment, the sending of the write_control IU 300originates the AWCE; whereas, according to other embodiments, the AWCEmay have already been originated before the target 104 received thewrite command IU. The frame header of the write_control IU 300 specifiesan OX_ID value selected by the target 104 for the AWCE, which in theexample of FIG. 8 has a value of 0x3333, and a RX_ID value, which in theexample of FIG. 8 has a value of 0xFFFF. The payload OX_ID field 302 ofthe write_control IU 300 specifies the WIOE OX_ID value, which in theexample of FIG. 8 has a value of 0x1111. The payload RX_ID field 304 ofthe write_control IU 300 specifies a RX_ID assigned by the target 104for the WIOE that the initiator learns, which in the example of FIG. 8has a value of 0x2222. Preferably, the write_control IU 300 does notcause the Sequence Initiative of the AWCE to be transferred to theinitiator 102.

As described at block 504 of FIG. 5, in response to receiving thewrite_control IU 300, the initiator 102 sends the target 104 a data IUwhose payload includes the data requested by the received write_controlIU 300, namely the data specified in the relative_offset field 312 andtransfer_length field 314. The frame header of the data IU specifies theWIOE OX_ID value of 0x1111 and RX_ID value of 0x2222. Preferably, theinitiator 102 does not cause the Sequence Initiative to be transferredback to the target 104.

As described at block 704 of FIG. 7, when the target 104 is ready (e.g.,buffer space is available) to receive data specified by the writecommand IU, and realizing that it has not yet received all the dataspecified by the write command IU, the target 104 sends the initiator102 a second write_control IU 300 within the AWCE, as described at block704. The second write_control IU 300 is similar to the first; however,the relative_offset field 312 and transfer_length field 314 values areupdated to reflect the next portion of the data. The target 104 may sendthe write_control IU 300 concurrently with the sending of the data IU bythe initiator 102 since they are being sent on two different FCExchanges, namely the WIOE and the AWCE. The AWCE enables the target 104to request the data from the initiator 102 as soon as the target 104 isready (e.g., buffer space is available) to receive data without havingto wait for the initiator 102 to transfer the WIOE Sequence Initiativeto the target 104. This advantageously reduces or eliminates much of thelatency incurred by a conventional FCP write I/O operation as describedin more detail with respect to FIGS. 13 and 14. As described herein, theAWCE may be terminated during the write I/O operation (preferably viathe last write_control IU 300 sent by the target 104), as described withrespect to the embodiment of FIG. 10; alternatively, the AWCE may remainopen in order to service other write I/O operations, as described withrespect to the embodiment of FIG. 9.

As described at block 504 of FIG. 5, in response to receiving the secondwrite_control IU 300, the initiator 102 sends the target 104, within theWIOE, a second data IU whose payload includes the data requested by thereceived second write_control IU 300. In the example of FIG. 8, this isthe last data IU, so the initiator 102 causes the Sequence Initiative tobe transferred back to the target 104. It should be understood thatalthough only two data IUs and two write_control IUs 300 are sent in theexample of FIG. 8, more than two could be sent in other FCP write I/Ooperations. Furthermore, it should be understood that because of theasynchronous nature of the target 104 sending the write_control IUs 300within the AWCE, the write_control IUs 300 could actually get ahead ofthe data IUs if buffer space quickly became available within the target104 because, for example, the target 104 freed a data buffer associatedwith another initiator 102 or with a different write I/O operation. Forexample, the target 104 could have started sending the secondwrite_control IU 300 before any of the data IU frames arrived, or evenbefore they were sent, as described with respect to FIG. 14.

In response to receiving the last data IU from the initiator 102, whichtransfers Sequence Initiative to the target 104, the target 104 sendsthe initiator 102 a response IU (FCP_RSP IU), within the WIOE, thatincludes the status of the write I/O operation, including the SCSIstatus associated with the SCSI command that was included in the writecommand IU. The target 104 terminates the WIOE via the response IU,namely by setting the Last_Sequence bit of the F_CTL field to one in theresponse IU.

As may be observed from FIG. 8, the employment of the AWCE effectivelyallows pipelining of the communication between the initiator 102 andtarget 104 to accelerate the generation of the data transfer request IUsrelative to the conventional FCP approach, thereby reducing, and in somecases eliminating, write I/O operation latency.

Referring now to FIG. 9, a flow diagram illustrating one embodiment ofthe manner in which an initiator 102 and a target 104 perform multiplewrite I/O operations is shown. According to the embodiment of FIG. 9,the initiator 102 and target 104 perform multiple write I/O operationsusing a single AWCE. Flow begins at block 902.

At block 902, the initiator 102 and target 104 perform a first write I/Ooperation using a first WIOE and an AWCE, such as the example write I/Ooperation described in FIG. 8. The AWCE is not terminated. Flow proceedsto block 904.

At block 904, the initiator 102 and target 104 perform a second writeI/O operation using a second WIOE and the AWCE that was used to performthe first write I/O operation. The AWCE is not terminated. Flow proceedsto block 906.

At block 906, the initiator 102 and target 104 perform an Nth write I/Ooperation using an Nth WIOE and the AWCE that was used to perform thefirst N−1 write I/O operations. The target 104 terminates the AWCE, forexample in response to the initiator 102 logging out. Flow ends at block906.

Referring now to FIG. 10, a flow diagram illustrating an alternateembodiment of the manner in which an initiator 102 and a target 104perform multiple write I/O operations is shown. According to theembodiment of FIG. 10, the initiator 102 and target 104 perform multiplewrite I/O operations each of which employs its own respective AWCE. Flowbegins at block 1002.

At block 1002, the initiator 102 and target 104 perform a first writeI/O operation using a first WIOE and a first AWCE, such as the examplewrite I/O operation described in FIG. 8. Preferably, in the embodimentof FIG. 10, the target 104 originates the AWCE via the firstwrite_control IU 300 sent to the initiator 102 in response to receivingthe write command IU. Subsequently, the target 104 terminates the firstAWCE, for example after sending the last write_control IU 300 to requestthe last data of the write I/O operation. Flow proceeds to block 1004.

At block 1004, the initiator 102 and target 104 perform a second writeI/O operation using a second WIOE and a second AWCE. Subsequently, thetarget 104 terminates the second AWCE. Flow proceeds to block 1006.

At block 1006, the initiator 102 and target 104 perform an Nth write I/Ooperation using an Nth WIOE and an Nth AWCE. Subsequently, the target104 terminates the Nth AWCE. Flow ends at block 1006.

Referring now to FIG. 11, a block diagram illustrating the payload of awrite_control IU 1100 according to an alternate embodiment is shown. Thewrite_control IU 1100 payload includes a num_elements field 1102 and aplurality of wc_info fields 1104, denoted as an array of elements [0],[1] and so forth to [N−1]. As shown, each wc_info field 1104 includessub fields that correspond to the fields of the write_control IU 300 ofFIG. 3, namely the OX_ID field 302, the RX_ID field 304, theoperation_type field 306, the sequence _initiative field 308, therelative_offset field 312 and the transfer_length field 314. The valueof the num_elements field 1102 indicates the number of valid wc_info1104 fields. Each wc_info field 1104 specifies a WIOE via the OX_IDfield 302 and RX_ID field 304. Also, each wc_info field 1104 specifiesthe data being requested within the specified WIOE via therelative_offset 312 and transfer_length 314 fields when theoperation_type field 306 indicates a request_dataout_transfer. Finally,each wc_info field 1104 specifies whether the initiator 102 is totransfer the WIOE Sequence Initiative to the target 104 via thecorresponding data IU.

Referring now to FIG. 12, a block diagram illustrating the payload of awrite_control IU 1200, which is an enhanced FCP_XFER_RDY IU, accordingto an alternate embodiment is shown. The write_control IU 1200 payloadincludes the conventional FCP_DATA_RO 1204 and FCP_BURST_LEN 1206 fieldsfor specifying the data being requested, which perform a functionsimilar to the relative_offset 312 and transfer_length 314 fields of thewrite_control IU 300 of FIG. 3. However, the write_control IU 1200 ofFIG. 12, that is, the enhanced FCP_XFER_RDY IU, also includes an AWCEflag 1208, an OX_ID field 1212, a RX_ID field 1214, and a sequence_initiative field 1216, all of which are included in bits of the payloadof the write_control IU 1200 that are reserved by the FCP-4 standard inthe payload of a conventional FCP_XFER_RDY IU. The OX_ID field 1212, theRX_ID field 1214, and the sequence _initiative field 1216 functionsimilarly to their counterparts in the write_control IU 300 of FIG. 3,namely the OX_ID field 302, the RX_ID field 304, and the sequence_initiative field 308 of the write_control IU 300. If the AWCE flag 1208is set to one, then the initiator 102 interprets the write_control IU1200 frame as an enhanced FCP_XFER_RDY IU, that is, the values of theOX_ID field 1212, the RX_ID field 1214, and the sequence _initiativefield 1216 are valid and should be used. However, if the AWCE flag 1208is cleared to zero, then the initiator 102 interprets the frame as aconventional FCP_XFER_RDY IU, that is, the values of the OX_ID field1212, the RX_ID field 1214, and the sequence _initiative field 1216 arenot valid and should not be used.

Referring now to FIG. 13, a timing diagram illustrating latenciesassociated with a conventional FCP write I/O operation such as describedin FIG. 2 is shown. Within FIGS. 13 and 14, various latencies aredenoted with capital letters within square brackets, such as “[A].” Asmay be observed from FIG. 13, the various latencies associated with theconventional FCP write I/O operation are incurred sequentially such thatthey add up to the total latency of the FCP write I/O operation. Inparticular, the processing latency [B] required for the target 104 todetect that the FC Exchange Sequence Initiative has been transferred toit and to begin transmitting the second FCP_XFER_RDY IU cannot startuntil the last frame of the FCP_DATA IU has been received, that is, thetransmission latency [A] of the first FCP_DATA IU has been incurred.Similarly, the transmission latency [C] associated with the secondFCP_XFER_RDY IU cannot start until the last frame of the first FCP_DATAIU has been received, that is, until the transmission latency [A] of thefirst FCP_DATA IU has been incurred and the target 104 processing time[B] has been incurred. Furthermore, the processing latency [D] requiredfor the initiator 102 to detect that the FC Exchange Sequence Initiativehas been transferred to it and to begin transmitting the second FCP_DATAIU cannot start until the second FCP_XFER_RDY IU has been received, thatis, until the transmission latency [C] of the second FCP_XFER_RDY IU hasbeen incurred. Similarly, the transmission latency [E] associated withthe second FCP_DATA IU cannot start until the second FCP_XFER_RDY IU hasbeen received, that is, until the transmission latency [C] of the secondFCP_XFER_RDY IU has been incurred and the initiator 102 processing time[D] has been incurred.

Referring now to FIG. 14, a timing diagram illustrating latenciesassociated with a FCP write I/O operation such as described in FIG. 8 isshown. The lines illustrating transmission of the write_control IUs 300within the AWCE are shown as dotted lines in FIG. 14, whereas the linesillustrating transmission of the FCP_CMND, FCP_DATA and FCP_RSP IUswithin the WIOE are shown as solid lines. As may be observed from FIG.14, the total latency of the FCP write I/O operation employing the AWCEis potentially greatly reduced relative to the convention write I/Ooperation. First, because the target 104 does not have to wait forSequence Initiative to be transferred to it by the data IU in order torequest the next data transfer from the initiator 102, some or all ofthe target 104 processing latency [F] from detecting that buffer spacehas become available for the second data IU can be overlapped with, orhidden behind, the transmission latency [J] of the first data IU. Infact, in some cases as in the example of FIG. 14, the target 104processing time [F] may even occur during the first write_control IU 300transmission time and potentially even before the initiator 102 beginsthe first data IU transmission. Similarly, some or all of thetransmission time [G] of the second write_control IU 300 may be hiddenbehind the transmission latency of the first data IU. These latencyreductions or eliminations are made possible by the full-duplexcommunications afforded by the AWCE as described above. Second, becausethe initiator 102 does not have to wait for Sequence Initiative to betransferred to it by the data request IU in order to send the seconddata IU, some or all of the initiator 102 processing time [H] fromreceiving the second write_control IU 300 can be overlapped with, orhidden behind, the transmission latency [J] of the first data IU. Infact, in some cases as in the example of FIG. 14, the initiator 102processing time [H] may even occur during the first data IU transmissiontime [J] and potentially even before the initiator 102 begins the firstdata IU transmission. Similarly, some of the transmission time [K] ofthe second data IU may be hidden behind the transmission latency [J] ofthe first data IU. Again, these latency reductions or eliminations aremade possible by the full-duplex communications afforded by the AWCE.

It should be understood that FIG. 14 illustrates a relatively best casescenario and the amount of latency reduction enjoyed by other FCP writeI/O operations may vary depending upon the conditions present, such asthe time in which buffers become available, times associated withprocessing other write I/O operations, and the availability oftransmission opportunity and bandwidth on the FC fabric 106.

It should be understood that the times shown in FIGS. 13 and 14 are notto scale and are not intended to show actual times, but are insteadprovided to illustrate the dependencies between the various FC frametransmission times and initiator 102 and target 104 processing times andthe latencies induced by the dependencies.

Although embodiments have been described in which an asynchronous writecontrol FC Exchange has been employed to reduce write I/O operationlatency with respect to SCSI as the FC-4 upper level protocol (FCP),other embodiments are contemplated in which an asynchronous FC exchangemay be employed to reduce write I/O latency with respect to other FC-4upper level protocols.

Furthermore, although different write_control IU embodiments have beendescribed, it should be understood that other embodiments may beemployed, such as different FC frame types or values includingenhancements of existing FC standard-defined frames or new FC frameswhose types or values are currently reserved by the FC standards andleft open for future expansion, so long at the target 104 is able torequest data transfers from the initiator 102 within a different FCExchange than the FC Exchange within which the initiator 102 sends thedata IUs to the target 104.

Additionally, although embodiments have been described in which the FCPinitiators and targets communicate via a FC fabric topology, otherembodiments are contemplated in which the FCP initiators and targetscommunicate via a FC arbitrated loop topology or FC point-to-pointtopology using the AWCE scheme described herein.

Furthermore, although embodiments have been described in which thetarget 104 constantly holds the Sequence Initiative for the AWCE, otherembodiments are contemplated in which the write_control IU transfers theAWCE Sequence Initiative to the initiator 102 and the initiator 102sends an “ACK” IU in response to transfer the AWCE Sequence Initiativeback to the target 104. The ACK IU may be another write_control IU, forexample one in which the operation_type field 306 has a distinctivevalue to indicate that the initiator 102 is acknowledging receipt of therequest_dataout_transfer operation_type 306 write_control IU and is nowready to receive another request_dataout_transfer write_control IU fromthe target 104. This embodiment provides a mechanism for the initiator102 to pace the target 104. According to an expansion of thisembodiment, the initiator 102 and target 104 establish a credit ofwrite_control IUs, such that the target 104 is allowed to send only theestablished credit number of write_control IUs to the initiator 102within the AWCE before it must wait for the initiator 102 to replenishits credit of write_control IUs. In this manner, the initiator 102 isenabled to impose flow control upon the target 104 at the write_controlIU level. Still further, embodiments are contemplated in which theinitiator 102 is not required to send the credit replenishments withinthe AWCE. In these embodiments, the initiator 102 and target 104 detectsupport for the AWCE feature and establish a credit for N outstandingwrite_control IUs. The N credits may apply only to a single FCP writeI/O operation, or the credits may apply across all FCP write I/Ooperations between the initiator 102 N_port and the target 104 N_port.Subsequently, the target 104 sends write_control IUs within the AWCE anddecrements its credit value, which may eventually reach zero, at whichtime the target 104 must stop sending write_control IUs. Eventually, theinitiator 102 sends the target 104 a credit replenishment to the target104, perhaps in the WIOE originated by the initiator 102, in response towhich the target 104 increments its credit value and resumes sendingwrite_control IUs, assuming it was blocked due to a zero credit value.

Finally, embodiments are contemplated in which the WIOE is effectivelytreated as unidirectional by the initiator 102 and target 104. Morespecifically, the target 104 firmware effectively ignores the SequenceInitiative bit of the F_CTL field of the FC frame headers of theFCP_CMND and FCP_DATA IUs even though they may be set to one to transferthe Sequence Initiative to the target 104. This embodiment may beparticularly useful in the case where a target 104 includes hardwarestate machines that would be cost prohibitive to modify, but upon whichhardware it is desirable to implement the full-duplex communicationscheme described herein. In such embodiments, the target 104 may send awrite_control IU 300 with a distinctive operation_type 306 value andpayload equivalent to the FCP_RSP IU within the AWCE as a response IU;alternatively, the target 104 firmware could recognize that the last ofthe data for the write I/O operation was received in a data IU andreceive the WIOE Sequence Initiative in order to send the FCP_RSP IUwithin the WIOE.

Embodiments described herein include the following potential advantages.First, employing the full-duplex communication between initiator andtarget afforded by the dual WIOE and AWCE FC Exchanges may reduce thelatency associated with a FCP write I/O operation relative toconventional systems that perform a FCP write I/O operation on a singlehalf-duplex exchange. Second, as a result of these latency reductions,which are discussed in more detail above, the target 104 may be enabledto request the data transfers (via the write_control IUs 300) in smallerchunks without incurring latency penalties that would be incurred by sodoing in a conventional FCP write I/O operation. This has the advantageof potentially affording higher performance with lower contiguous bufferrequirements in the target 104.

While various embodiments of the present invention have been describedherein, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant computing arts that various changes in form and detail canbe made therein without departing from the scope of the invention. Forexample, software can enable the function, fabrication, modeling,simulation, description and/or testing of the apparatus and methodsdescribed herein. This can be accomplished through the use of generalprogramming languages (e.g., C, C++), hardware description languages(HDL) including Verilog HDL, VHDL, and so on, or other availableprograms. Such software can be disposed in any known non-transitorycomputer usable medium such as magnetic tape, semiconductor, magneticdisk, or optical disc (e.g., CD-ROM, DVD-ROM, etc.), a network, or wireline, or other communications medium. Embodiments of the apparatus andmethod described herein may be included in an integrated circuit.Additionally, the apparatus and methods described herein may be embodiedas a combination of hardware and software. Thus, the present inventionshould not be limited by any of the exemplary embodiments describedherein, but should be defined only in accordance with the followingclaims and their equivalents. Finally, those skilled in the art shouldappreciate that they can readily use the disclosed conception andspecific embodiments as a basis for designing or modifying otherstructures for carrying out the same purposes of the present inventionwithout departing from the scope of the invention as defined by theappended claims.

We claim: 1-20. (canceled)
 21. A system, comprising: a Fibre Channel(FC) Protocol for SCSI (FCP) target device comprising at least aprocessor, wherein the FCP target device is operable to: send one ormore FC write control information units (IUs) to a FCP initiator devicewithin a first FC exchange to request a transfer of data associated witha FCP write command IU previously sent to the FCP target device by theFCP initiator device within a second FC exchange; wherein the FC writecontrol IUs are sent without the FCP target device first receiving fromthe FCP initiator device a sequence initiative of the second FCexchange.
 22. The system of claim 21, wherein the FCP target device isoperable to send the one or more write control IUs within the first FCexchange concurrently with the FCP initiator device sending one or moreFCP data IU sequences within the second FC exchange to the FCP targetdevice.
 23. The system of claim 21, wherein the FCP target device isoperable to send a second one or more write control IUs to the FCPinitiator device within the first FC exchange to request a transfer ofdata associated with a second FCP write command IU previously sent tothe FCP target device by the FCP initiator device within a third FCexchange; wherein the first FC exchange is distinct from the third FCexchange.
 24. The system of claim 21, wherein the payload of each writecontrol IU of the one or more write control IUs comprises one or moreof: an originator exchange identifier (OX_ID) value with which the FCPinitiator originated the second FC exchange, a responder exchangeidentifier (RX_ID) value assigned by the FCP target for the second FCexchange, and an indicator that instructs the FCP initiator devicewhether to hold or transfer the sequence initiative of the second FCexchange to the FCP target device.
 25. The system of claim 21, whereineach write control IU of the one or more write control IUs comprises anew FCP IU previously undefined by the FCP-4 standard.
 26. The system ofclaim 21, wherein each write control IU of the one or more write controlIUs comprises an enhanced FCP transfer ready (FCP_XFER_RDY) IU, whereinthe payload of the write control IU is located in bits previouslyreserved by the FCP-4 standard.
 27. The system of claim 21, wherein theone or more write control IUs hold a sequence initiative of the first FCexchange.
 28. A system, comprising: a Fibre Channel (FC) Protocol forSCSI (FCP) initiator device comprising at least a processor, wherein theFCP initiator device is operable to: receive one or more FC writecontrol information units (IUs) from a FCP target device within a firstFC exchange requesting a transfer of data associated with a FCP writecommand IU previously sent to the FCP target device by the FCP initiatordevice within a second FC exchange; wherein the FC write control IUs arereceived without the FCP initiator device first sending a sequenceinitiative of the second FC exchange.
 29. The system of claim 28,wherein the FCP initiator device is configured to receive the one ormore write control IUs within the first FC exchange concurrently withsending one or more FCP data IU sequences within the second FC exchangeto the FCP target device.
 30. The system of claim 28, wherein the FCPinitiator device is operable to receive a second one or more writecontrol IUs from the FCP target device within the first FC exchangerequesting a transfer of data associated with a second FCP write commandIU previously sent to the FCP target device by the FCP initiator devicewithin a third FC exchange; wherein the first FC exchange is distinctfrom the third FC exchange;
 31. The system of claim 28, wherein thepayload of each write control IU of the one or more write control IUscomprises one or more of: an originator exchange identifier (OX_ID)value with which the FCP initiator device originated the second FCexchange, a responder exchange identifier (RX_ID) value assigned by theFCP target device for the second FC exchange, an indicator thatinstructs the FCP initiator device whether to hold or transfer thesequence initiative of the second FC exchange to the FCP target device.32. The system of claim 28, wherein each write control IU of the one ormore write control IUs comprises a new FCP IU previously undefined bythe FCP-4 standard.
 33. The system of claim 28, wherein each writecontrol IU of the one or more write control IUs comprises an enhancedFCP transfer ready (FCP_XFER_RDY) IU, wherein the payload of the writecontrol IU is located in bits previously reserved by the FCP-4 standard.34. The system of claim 28, wherein the one or more write control IUshold a sequence initiative of the first FC exchange.
 35. The system ofclaim 28, wherein the FCP write command IU sent to the FCP target deviceby the FCP initiator device holds the sequence initiative of the secondFC exchange.
 36. A method, comprising: sending, by a Fibre Channel (FC)Protocol for SCSI (FCP) target device, one or more FC write controlinformation units (IUs) to an FCP initiator device within a first FCexchange to request a transfer of data associated with a FCP writecommand IU previously sent to the FCP target device by the FCP initiatordevice within a second FC exchange; wherein the FC write control IUs aresent without the FCP target device first receiving from the FCPinitiator device a sequence initiative of the second FC exchange. 37.The method of claim 36, comprising sending the one or more write controlIUs within the first FC exchange concurrently with the FCP initiatordevice sending one or more FCP data IU sequences within the second FCexchange to the FCP target device.
 38. The method of claim 36, whereineach write control IU of the one or more write control IUs comprises anew FCP IU previously undefined by the FCP-4 standard.
 39. The method ofclaim 36, wherein each write control IU of the one or more write controlIUs comprises an enhanced FCP transfer ready (FCP_XFER_RDY) IU, whereinthe payload of the write control IU is located in bits previouslyreserved by the FCP-4 standard.
 40. The method of claim 36, wherein theone or more write control IUs hold a sequence initiative of the first FCexchange.